Coreference Resolution Evaluation Based on Descriptive Specificity

This paper introduces a new evaluation method for the coreference resolution task. Considering that coreference resolution is a matter of linking expressions to discourse referents, we set our evaluation criteron in terms of an evaluation of the denotations assigned to the expressions. This criterion requires that the coreference chains identified in one annotation stand in a one-to-one correspondence with the coreference chains in the other. To determine this correspondence and with a view to keep closer to what human interpretation of the coreference chains would be, we take into account the fact that, in a coreference chain, some expressions are more specific to their referent than others. With this observation in mind, we measure the similarity between the chains in one annotation and the chains in the other, and then compute the optimal similarity between the two annotations. Evaluation then consists in checking whether the denotations assigned to the expressions are correct or not. New measures to analyse errors are also introduced. A comparison with other methods is given at the end of the paper. Identifying expressions which, in a text, denote the same discourse referent is usually considered a key process in automatic information extraction. However, the question of how to evaluate coreference resolution systems has sometimes been an issue: after the publication by Vilain et al. (1995) of a new evaluation scoring scheme for the Message Understanding Conferences, Popescu-Bellis et al. (1998) and Bagga et al. (1998) each proposed new evaluation methods. In this paper, we, in turn, propose a new evaluation method for the coreference resolution task. A coreference chain is defined by the property the expressions it contains have to denote a specific discourse referent (1). Our evaluation method so aims at evaluating coreference resolution with respect to this property, the problem being to evaluate whether the discourse referents associated with the expressions are the correct ones (2). From this setting of coreference resolution evaluation in terms of denotation assignment, one derives some constraints on the way two annotations should correspond; in particular, we observe that the fact that some expressions in a coreference chain are more specific to their referent than others has to be taken into account (3). The implementation of our evaluation method meets our requirements by computing the optimal similarity between the coreference chains in two annotations using a linear combination of Dice coefficients over some subsets of coreference chains (4). The recall and precision measures then express a comparison of two sets of denotation assignments. Three complementary measures for errors analysis are also proposed (5). Finally, we show how our evaluation method relates with existing ones (6).